Similarity preserving compressions of high dimensional sparse data

The rise of internet has resulted in an explosion of data consisting of millions of articles, images, songs, and videos. Most of this data is high dimensional and sparse. The need to perform an efficient search for similar objects in such high dimensional big datasets is becoming increasingly common. Even with the rapid growth in computing power, the bruteforce search for such a task is impractical and at times impossible. Therefore algorithmic solutions such as Locality Sensitive Hashing (LSH) are required to achieve the desired efficiency in search. Any similarity search method that achieves the efficiency uses one (or both) of the following methods: 1. Compress the data by reducing its dimension while preserving the similarities between any pair of data-objects 2. Limit the search space by grouping the data-objects based on their similarities. Typically 2 is obtained as a consequence of 1. Our focus is on high dimensional sparse data, where the standard compression schemes, such as LSH for Hamming distance (Gionis, Indyk and Motwani [7]), become inefficient in both 1 and 2 due to at least one of the following reasons: 1. No efficient compression schemes mapping binary vectors to binary vectors 2. Compression length is nearly linear in the dimension and grows inversely with the sparsity 3. Randomness used grows linearly with the product of dimension and compression length. We propose an efficient compression scheme mapping binary vectors into binary vectors and simultaneously preserving Hamming distance and Inner Product. Our schemes avoid all the above mentioned drawbacks for high dimensional sparse data. The length of our compression depends only on the sparsity and is independent of the dimension of the data. Moreover our schemes provide one-shot solution for Hamming distance and Inner Product, and work in the streaming setting as well. In contrast with the “local projection” strategies used by most of the previous schemes, our scheme combines (using sparsity) the following two strategies: 1. Partitioning the dimensions into several buckets, 2. Then obtaining “global linear summaries” in each of these buckets. We generalize our scheme for real-valued data and obtain compressions for Euclidean distance, Inner Product, and k-way Inner Product.